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Discrepant LSAT Subscores 



Abstract 

This study investigated the overall prevalence of discrepant LSAT subscores, their differential incidence for 
subgroups of examinees, and the psychometric properties of alternative measures of discrepant perfor- 
mance. Subscore differences, often very substantial ones, were frequent; statistically significant differences 
affected about a third of examinees; and significant and rare differences involved a tenth of test takers. The 
incidence of these discrepancies did not vary with the examinees' sex, ethnicity, familiarity with the LSAT, or 
the number or selectivity of the law schools to which the examinees were applying. But the prevalence was 
greater for examinees who had high total scores on the LSAT or were older, primarily reflecting these test- 
takers' deviantly poor performance on the Analytical Reasoning subtest. Reliability was appreciable for two 
of the three measures of observed differences but minimal for the more important measures of significant or 
significant and rare differences. Subscore discrepancies appear to have no viable role to play in interpreting 
examinees' LSAT performance. 

Unlike other tests employed in graduate admissions, the Law School Admission Test (LSAT) reports only a 
total score. In contrast, the Graduate Record Examination General Test (Conrad, Trisman, & Miller, 1977) pro- 
vides Verbal, Quantitative, and Analytical scores (and no total score), and the Graduate Management 
Admission Test (Hecht & Schrader, 1986) gives Verbal and Quantitative scores as well as a total score. 

A recent LSAS survey suggested that there was growing interest in the law school community about the po- 
tential usefulness of the LSAT subscores (Analytical Reasoning, Reading Comprehension, and Logical 
Reasoning) in admissions, especially for minority applicants, supplementing what is yielded by the test's 
total score. One approach to this issue is to identify examinees with markedly different performance on the 
test' s subscores. Previous research has studied examinees' discrepancies between their LSAT performance 
and their college record, as reflected in large differences between LSAT scores and imdergraduate GPAs, in 
an effort to determine the implications of these differences for the validity of the test and college record in 
predicting law school grades (Boldt, 1966, 1968). Similarly, examinees may exhibit discrepant performance 
on the various sections of the LSAT, with large differences in their subscores. These discrepancies may shed 
light on the examinees' overall test performance, useful in interpreting the LSAT total score and in making 
admissions decisions about these applicants. For instance, discrepant performance could reflect language dif- 
ficulties or atypical education. 

"Profile" or "scatter" analysis of discrepancies in the subtest scores on individually-administered intelli- 
gence tests, notably the Weschler Adult Intelligence Scale — Revised (WAIS-R; Wechsler, 1981), Wechsler 
Intelligence Scale for Children — Revised (WISC-R; Wechsler, 1974), and Stanford-Binet Intelligence Scale: 
Fourth Edition (Thorndike, Hagen, & Sattler, 1986), have long been employed in interpreting performance 
on these tests (e.g., Frank, 1983; Kaufman, 1979, 1990; Matarazzo, 1972; Sattler, 1988). For instance, differ- 
ences between the Verbal and Performance IQs on the WAIS-R or WISC-R are examined. Subtest scores on 
these tests are also compared, such as the difference between the highest and lowest scores for the entire set 
of subtests, as well as the difference between scores for each pair of subtests. 

Accordingly, the purpose of this study was to provide basic iifformation on discrepant test performance on 
the LSAT: the overall prevalence of discrepant performance as well as its differential incidence for relevant 
subgroups of examinees, and the psychometric properties of alternative measures of this characteristic. 



Method 



Sample and Test Form 

The sample consisted of the 39^30 examinees who took the LSAT (Form 2LSS13) at the December 1991 ad- 
ministration, had no test irregularities, had no handicap flag, and had not applied to all 15 Canadian law 
schools. This test administration was chosen because it used the current version of the test, it was large, and 
it had a substantial proportion of minority examinees. For the sample, 55.1% were male, 72.4% were Cauca- 
sian, the mean age was 26.4 years (SD = 6.51), and the mean LSAT score was 149.9 (SD = 9.2). 

Variables 

Discrepant subscores. A combined score was obtained for the two separately-timed Logical Reasoning sec- 
tions. The raw scores for this combined subtest and for each of the two other subtests were converted to 
standard scores. Corresponding standard scores for the halves of each subtest were also obtained. The 
halves were matched in content and psychometric characteristics. The content characteristics were the major 
categories used in test development. Analytical Reasoning categories are Ordering, Grouping and Set Mem- 
bership, and Mixed/ Other. Reading Compreheirsion categories are Science and Technology, Hunranities, 
Social Science, and Law. And Logical Reasoning categories are Conclusions; Techniques, Structures, Pat- 
terns, or Principles; Argument Flaws and Reasoning Errors; and Additional Evidence. The psychometric 
characteristics were the delta indexes of item difficulty and biserial correlatioirs of item discrimination, from 
the routine test analysis for a subsample of the examinees. 

Three kinds of discrepancy scores, adapted from those employed with intelligence tests (e.g., Kaufman, 

1990; Sattler, 1988), were obtained for each pair of Subscores: 

1 . An observed difference: The actual difference between a pair of subscores, retaining the direction of the 
difference. 

2. A significant difference: An observed difference that was statistically significant. Each pair of subscore dif- 
ferences was tested for significance, using the .05 significance level (two-tailed). The direction of the 
difference was retained. (The size of each pair of subscore differences required for a significant differ- 
ence appjears in Table 1.) 

3. A significant and rare difference: An observed difference that was both significant and infrequent. Each 
pair of subscore differences was appraised for its frequency in the sample. Subscore differences were 
identified that had a probability of occurrence of .05 or less (using the top and bottom 2.5% of the fre- 
quency distribution) and were also significant. The direction of the difference was retained. (The size of 
each pair of subscore differences required for a significant and rare difference appears in Table 1.) 

Two composite measures were also obtained from the discrepancy scores for the three pairs of subscores: 

1 . The number of significant differences, disregarding the direction of the differences. 

2. The number of significant and rare differences, disregarding the direction of the differences. 

The following variables were derived for the full subscores and the half subscores: 

1. Observed difference for Analytical Reasoning vs. Reading Comprehension. 

2. Observed difference for Analytical Reasoning vs. Logical Reasoning. 

3. Observed difference for Reading Comprehension vs. Logical Reasoning. 

4. Significant difference for Analytical Reasoning vs. Reading Comprehension. 

5. Significant difference for Analytical Reasoning vs. Logical Reasoning. 
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6. Significant difference for Reading Comprehension vs. Logical Reasoning. 

7. Number of significant differences. 

8. Significant and rare differences for Anal)^cal Reasoning vs. Reading Comprehension. 

9. Significant and rare differences for Anal)^cal Reasoning vs. Reading Comprehension. 

10. Significant and rare differences for Reading Comprehension vs. Logical Reasoning. 

11. Number of significant and rare differences. 



Table 1 

Size of Significant and Rare Differences in LSAT Subscores 



Subscore Pair 


Significant Difference 


Rare Difference 


Anal)^cal Reasoiring vs. 
Reading Comprehension 


>1.27, <-1.27 


>1.97, <-1.99 


Anal)^cal Reasoiring vs. 
Logical Reasoning 


>1.15, <-1.15 


>1.75, <-1.82 


Reading Comprehension vs. 
Logical Reasoiring 


■ >1.13, <-1.13 


>1.36, <-1.35 



Subgroups 

The subgroup variables in the analysis are listed below. They were derived from data on the registration 
form, answer sheet, and test records. Continuous variables were trichotomized to yield subsamples roughly 
equal in size. 

1. Sex. 

2. Age (17-22, 23-26, 27-72). 

3. Ethnicity (American Indian, Asian, Black, Canadian Aboriginal and Other, Caucasian, Chicano, Hispanic, 
and Puerto Rican). 

4. English language dominance (English or other). 

5. English fluency (fluent or not fluent). 

6. LSAT total score (scaled score form; 120-146, 147-154, 155-180). 

7. Number of LSATs taken (1, 2, 3-18). 

8. LSAT Preparation (any preparation or no preparation).^ 

9. Number of law school applications (1-2, 3-6, 7-129). 

10. Selectivity of law schools (median for the mean LSAT scores in 1990-1991 for admitted students in 1990- 
1991 in each of the law schools to which the exanunee is applying; 16-36, 37-38, 39-45).^ 
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Analysis 



Prevalence analysis. Differences within the various subgroups (e.g., male vs. female) in the frequencies for 
each of the six categorical discrepancy variables for the full subscores (e.g., significant difference for Analyti- 
cal Reasoning vs. Reading Comprehension, significant and rare difference for Analytical Reasoning vs. 
Reading Comprehension) were assessed by tests. And differences within the various subgroups in the 
means for each of the continuous discrepancy variables for the full subscores (e.g., observed difference for 
Analytical Reasoning vs. Reading Comprehension, number of significant differences, number of significant 
and rare differences) were assessed by one-way analysis of variance. 

Because of the large sample size, both statistical and practical significance were considered in evaluating the 
results. The .01 level was used throughout in view of the sample size. A minimum effect size was used that 
accoimted for 1% of the variance: a w of .10 for x^ analyses and an T] of .10 for analyses of variance (Cohen, 
1988). This size is commonly considered to be a "small" effect from the standpoint of practical significance 
(Cohen, 1988). 

Reliability analysis. The agreement between the categorical discrepancy variable for the corresponding 
pairs of half subscores (e.g., significant difference for Analj^cal Reasoning — Half 1 vs. Reading Comprehen- 
sion — Half 1 with significant difference for Analytical Reasoning — Half 2 vs. Reading Comprehension — Half 
2) for the total sample was assessed by the product-moment correlation between the two trichotomous vari- 
ables. (These variables were dummy coded: The codes for the significant difference variables were +1 = 
significant — positive, 0 = not significant, and 1 = significant — negative; the codes for the significant and rare 
difference variables were: +1 = significant and rare — positive, 0 = not significant and rare, and -1 = signifi- 
cant and rare — ^negative.) 

The agreement between the continuous discrepancy variables for the corresponding pairs of half subscores 
(e.g., observed difference for Analytical Reasoning — Half 1 vs. Reading Comprehension — Half 1 with ob- 
served differences for Analytical Reasoning — ^Half 2 vs. Reading Comprehension — Half 2; number of 
significant differences — ^Half 1 with number of significant differences — Half 2) for the total sample was as- 
sessed by the product-moment correlation between the two continuous variables. 

It should be noted that these analyses yield imderestimates of reliability for the categorical discrepancy vari- 
ables and the composite measures for them because these variables are derived tom half subscores, not full 
subscores. Standard corrections for double length are not applicable, for these adjustments are for continu- 
ous scores that are additive, but the present variables use categorizations that are not additive. 

Inteicorrelation analysis. The product-moment intercorrelations of the categorical discrepancy variables for 
the full subscores were computed for the total sample. (The variables were dummy coded, wi& the same 
codes used in the reliability analysis.) Note that these correlations are inflated because the same subscores 
are used in different discrepancy variables (e.g., observed difference for Analytical Reasoning vs. Reading 
Comprehension with observed difference for Analytical Reasoning vs. Logical Reasoning). 
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Results 



Prevalence 

Observed differences. The frequency distributions for the three observed difference variables appear in 
Table 2 and in Figures 1 to 3. All the distributions were symmetrical, with roughly equal numbers of positive 
and negative score differences. The percentages of subscore differences of one point or more (disregarding 
the direction of the difference)— a difference of at least one standard score— ranged from 14.7% for Reading 
Comprehension vs. Logical Reasoning to 31.2% for Analytical Reasoning vs. Reading Comprehension. 

The means and standard deviations for these three variables in the subgroups are reported in Table 3, and 
the analyses of variance for the subgroups are summarized in Table 4. Several differences within subgroups 
in these variables were significant, statistically (p <.01) and practically ( t] >.10). All three variables were sig- 
nificant for the LSAT total score. In each case, the observed difference score was larger for the lower-scoring 
examinees (favoring Analytical Reasoning over Reading Comprehension or Logical Reasoning, and favoring 
Reading Comprehension over Logical Reasoning). 

Two of the variables were sigiuficant for age: Analytical Reasoning vs. Reading Comprehension, and Analyt- 
ical Reasoning vs. Logical Reasoning. In each case, the observed difference score was lower for the older 
examinees (favoring Reading Comprehension and Logical Reasoning over Analytical Reasoiung). 



Table 2 

Incidence of Obsenred Differences for Each Pair of LSAT Subscores 



Observed Difference 
Score 


Analytical Reasoning 
vs. 

Reading Comprehension 
(%) 


Analytical Reasoning 
vs. 

Logical Reasoning 
(%) 


Reading Comprehension 
vs. 

Logical Reasoning 
(%) 




“ 3.00+ 


.1 


.0 


.0 




2.50- 2.99 


.4 


2 


.0 




2.00- 2.49 


1.7 


1.0 


2 


+ 


1.50- 1.99 


4.3 


33 


13 




1.00- 1.49 


9.1 


83 


6.0 




.50 - .99 


153 


15.8 


15.6 




_ .01 - .49 


18.8 


213 


26.9 




“ .00 - .49 


19.6 


21.2 


263 




.50- .99 


15.0 


14.6 


16.3 




1.00- 1.49 


9.1 


83 


5.6 


— 


1.50- 1.99 


4.2 


3.6 


1.4 




2.00- 2.49 


1.7 


12 


.2 




2.50- 2.99 


5 


.3 


.0 




_ 3.00+ 


.1 


.1 


.0 
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Table 3 

Means and Standard Deviation of Observed Differences for Each Pair of LSAT Subscores 







Anal}^cal Reasoning vs. Anal}^cal Reasoning vs. 
Reading Comprehension Logical Reasoning 


Reading Comprehension 
vs. Logical Reasoning 


Subgroup 


N 


Mean 


m 




Mean 


SD 


Mean 


SD 










Sex 










Male 


21,762 


-.04 


1.00 




-.09 


.91 


-.05 


.70 


Female 


17,622 


.05 


.98 




.11 


.88 


.06 


.68 










Age 










17-22 


14,202 


.13 


.99 




.13 


.88 


.00 


.70 


23-26 


11,910 


.12 


.98 




.07 


.89 


-.04 


.69 


27-72 


13,397 


-.25 


.96 




-21 


.89 


.04 


.68 








Ethnicity 








American 

Indian 


276 


-.07 


.99 




-.01 


.93 


.06 


.72 


Asian 


2,377 


J21 


1.03 




28 


.90 


.07 


.70 


Black 


4,454 


.01 


.88 




.09 


.80 


.08 


.66 


Canadian 
Aboriginal 
and Other 


1,139 


-.01 


.98 




.02 


.89 


.04 


.69 


Caucasian 


28,503 


-.02 


1.01 




-.05 


.92 


-.03 


.69 


Chicano 


626 


-.05 


1.00 




.04 


.86 


.09 


.71 


Hispanic 


1,229 


-.01 


.94 




.10 


.86 


.11 


.66 


Puerto Rican 


753 


.07 


.87 




.18 


.78 


.10 


.63 








English Language Dominance 








English 


35,661 


-.01 


.99 




.00 


.90 


.00 


.69 


Other 


2,036 


.17 


.97 




.20 


.85 


.03 


.68 








English Fluency 








Fluent 


36,671 


.00 


.99 




.00 


.90 


.00 


.69 


Not Fluent 


348 


.29 


.93 




25 


.78 


.05 


.66 
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Table 3 (continued) 







Analytical Reasoning vs. 
Reading Comprehension 


Analytical Reasoning vs. 
Logical Reasoning 


Reading Comprehension 
vs. Logical Reasoning 


Subgroup 


N 


Mean 


SD 


Mean 


SD 


Mean 


SD 








LSAT Total Score 








120 - 146 


13,813 


.11 


.88 


21 


.80 


.10 


.66 


147-154 


13,174 


.02 


1.06 


-.01 


.95 


-.03 


.73 


155 - 180 


12,543 


-.15 


1.02 


-23 


.90 


-.07 


.66 








Number of LSATs Taken 








1 


29,035 


-.05 


.99 


-.05 


.90 


.00 


.69 


2 


8,675 


.13 


.99 


.12 


.91 


-.01 


.70 


3-18 


1,820 


.15 


.98 


20 


.87 


.04 


.69 








LSAT Preparation 








Preparation 


34,668 


.01 


.99 


.01 


.90 


.00 


.69 


No 

Preparation 


1,648 


-.18 


.98 


-.15 


.90 


.04 


.67 






Number of Law School Applications 








1-2 


9,617 


-.10 


.97 


-.08 


.90 


.02 


.68 


3-6 


10,063 


.00 


1.00 


.02 


.90 


.01 


.69 


7-129 


8,319 


.13 


1.00 


.08 


.90 


-.05 


.69 








Selectivity of Schools 








16-36 


7,511 


.06 


.99 


.07 


.90 


.01 


.70 


37-38 


8,044 


.03 


1.01 


.02 


.91 


.00 


.70 


39-45 


11,225 


-.05 


.99 


-.06 


.89 


-.01 


.68 
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Table 4 

Analyses of Variance of Subgroup Differences in Observed Differences for Each Pair of LSAT Subscores 







Analytical Reasoning vs. 
Reading Comprehension 


Analytical Reasoning vs. 
Logical Reasoning 


Reading Comprehension 
vs. Logical Reasoning 


Source 


df 


Mean 

Square 


F 


i\ 


Mean 

Square F 


i\ 


Mean 

Square 


F 


i\ 












Sex 










Between 


1 


71.28 


72.26** 


.04 


361.65 449.06** 


.11 


111.81 


236.66** 


.08 


Within 


39,382 


.99 






.81 




.47 
















Age 










Between 


2 


619.55 


647.01** 


.18 


444.33 560.83** 


.17 


23.28 


49.11** 


.05 


Within 


39,506 


.96 






.79 




.47 
















Ethnicity 










Between 


7 


17.22 


17.47** 


.06 


46.53 57.68** 


.10 


14.09 


29.81** 


.07 


Within 


34,349 


.99 






.81 




.47 














English Language Dominance 










Between 


1 


58.86 


59.65** 


.04 


78.96 9733** 


.05 


1.47 


3.11 


.01 


Within 


37,695 


.99 






.81 




.47 














English Fluency 










Between 


1 


29.21 


29.55** 


.03 


40.68 49.98** 


.04 


.95 


1.99 


.01 


Within 


37,017 


.99 






.81 




.48 














LSAT Total Score 










Between 


2 


239.69 


245.36** 


.11 


632.42 807.72** 


.20 


104.47 


22228** 


.11 


Within 


39,527 


.98 






.78 




.47 














Number of LSATs Taken 










Between 


2 


140.62 


143.21** 


.08 


134.69 166.61** 


.09 


2.42 


5.09** 


.02 


Within 


39,527 


.98 






.81 




.48 














LSAT Preparation 










Between 


1 


58.70 


59.46** 


.04 


39.10 47.92** 


.04 


1.98 


4.19 


.01 


Within 


36,314 


.99 






.82 




.47 












Number of Law School Applications 










Between 


2 


122.64 


125.19** 


.09 


64.45 79.49** 


.08 


11.29 


23.69** 


.04 


Within 


27,996 


.98 






.81 




.48 














Selectivity of Law Schools 










Between 


2 


31.93 


32.26** 


.05 


42.49 52.28** 


.06 


.77 


1.61 


.01 


Within 


26,777 


.99 






.81 




.48 







Note. An Ti of .10 is a "small" effect (Cohen, 1988). 
**^<.01. 




Figure 1 

Distribution for Analytical Reasoning vs. Reading Comprehension observed difference score. 




Figure 2 

Distribution for Analytical Reasoning vs. Logical Reasoning observed difference score. 




Figure 3 

Distribution for Reading Comprehension vs. Logical Reasoning observed difference score. 




One vciriable. Analytical Reasoning vs. Logical Reasoning, was significant for both sex and ethnicity. For sex, 
the observed difference score was higher for females (favoring Analytical Reasoning), and for ethnicity, the 
score was higher for Asians. 

In brief, observed differences between subscores were substantial. These differences varied with the LSAT 
total score and, to a lesser extent, with age, sex, and ethnicity. Most of these subgroup relations involved 
pairs of subscores that included Analytical Reasoning. 

Significant differences. The frequencies of a significant difference for each pair of subscores in the total sam- 
ple are reported in Table 5. The ftequency of positive and negative differences was roughly equal. The total 
number of significemt differences (positive or negative) ranged from 102% for Reading Comprehension vs. 
Logical Reasoning to 203% for Analytical Reasoning vs. Logical Reasoning. The frequencies for the number 
of significant differences for the three pairs of subscores in the total sample are reported in Table 6; 34.0% of 
the sample had one or more significant differences, and the mean was (SD = .76). 

The frequencies of a significant difference for each pair of subscores in the subgroups are reported in Table 7, 
and the analyses for the subgroups are summarized in Table 8. Several differences within subgroups in 
these variables were significant, statistically (p < .01) and practically (w > .10). Two of the variables were 
significcmt for age: Analytical Reasoning vs. Reading Comprehension, and Analytical Reasoning vs. Logical 
Reasoning. For both variables, the largest discrepancies occurred for older examinees: For Analytical Rea- 
soning vs. Reading Comprehension, fewer positive differences (favoring Analytical Reeisoning) and more 
negative differences; emd for Analytical Reasoning vs. Logical Reasoning, more negative differences (favor- 
ing Logical Reasoning). 




These two variables were also significant for the LSAT total score. For Analytical Reasoning vs. Reading 
Comprehension, the largest discrepancies involved fewer negative differences (favoring Reading Compre- 
hension) for low-scoring examinees. For Analytical Reasoning vs. Logical Reasoning, the largest 
discrepancies involved fewer negative differences (favoring Logical Reasoning) for low-scoring examinees 
and more negative differences for high-scoring examinees. 

The mean number of significant differences for the three pairs of subscores for the subgroups are reported in 
Table 9, and the analyses of variance for the subgroups are summarized in Table 10. Only one of the differ- 
ences within subgroups for this variable was significant, statistically and practically. For the LSAT total 
score, the mean for this discrepancy measure was smaller for the low-scoring examinees than for the other 
examinees. 

In short, significant differences on the pairs of subscores were common, affecting about a third of the exami- 
nees, especially when the Analytical Reasoning subscore was involved. These differences varied with the 
LSAT total score and age. 



Table 5 

Incidence of Significant and Rare Differences for Each Pair of LSAT Subscores 



Subscoxe Pair 


Significant Difference 


Rare Difference 




(-%) 


(+%) 


(-%) 


(+%) 


Analytical Reasoning vs. 
Reading Comprehension 


9.8 


9.8 


2.5 


2.5 


Analytical Reasoning vs. 
Logical Reasoning 


10.3 


10.0 


2.5 


2.5 


Reading Comprehension vs. 
Logical Reasoning 


5.1 


5.1 


.25 


2.5 



Frequency 



Table 6 

Incidence of Significant and Rare Differences for All Pairs of LSAT Subscores 



Significant Difference 

(%) 



Rare Difference 

(%) 



0 

1 

2 

3 



66.1 


88.1 


18.1 


8.8 


155 


3.1 


.4 


.0 
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Table 7 

Incidence of Significant and Rare Differences for Each Pair of LSAT Subscores for Subgroups 



Significant Difference Rare Difference 

ARvs.RC® ARvs.LRb RCvs-LR*" ARvs.RC® ARvs.LR^ RCvs-LR*" 



Subgroup 


N 


(-%) 


(+%) 


(-%) 


(+%) 


(-%) 


(+%) 


(-%) 


(+%) 


(-%) 


(+%) 


(-%) 


(+%) 


Sex 


Male 


21,762 


10.7 


9.5 


12.3 


8.7 


5.9 


4.4 


2.8 


2.4 


3.2 


2.2 


2.9 


2.2 


Female 


17,622 


8.7 


10.2 


7.8 


11.6 


4.2 


6.0 


2.2 


2.6 


1.6 


2.9 


2.0 


2.9 


Age 


17-22 


14,202 


7.6 


12.2 


7.3 


12.3 


5.4 


5.3 


1.8 


3.4 


1.5 


3.5 


2.7 


2.8 


23-26 


11,910 


7.6 


11.9 


9.0 


11.4 


5.9 


4.4 


1.8 


2.9 


1.9 


2.9 


3.1 


2.2 


27-72 


13,397 


14.0 


5.3 


14.7 


6.2 


4.1 


5.5 


3.9 


1.2 


4.1 


1.2 


1.8 


2.5 


Ethnicity 


American Indian 


276 


12.7 


9.4 


10.1 


10.9 


6.2 


5.8 


2.5 


1.4 


2.5 


1.4 


3.6 


2.9 


Asian 


2,377 


7.6 


14.7 


6.2 


15.9 


4.1 


6.3 


2.3 


3.9 


1.1 


5.0 


2.1 


3.0 


Black 


4,454 


7.0 


7.4 


6.0 


8.9 


3.2 


5.9 


1.6 


1.4 


1.1 


1.6 


1.3 


2.8 


Canadian 
Aboriginal and 


Other 


1,139 


9.4 


9.4 


9.0 


10.4 


4.6 


5.2 


2.3 


2.5 


2.9 


2.3 


1.8 


2.5 


Caucasian 


28,503 


10.5 


9.9 


11.6 


9.6 


5.7 


4.8 


2.8 


2.6 


2.9 


2.5 


2.8 


2.3 


Chicano 


626 


11.3 


8.6 


8.5 


9.9 


3.2 


7.4 


2.6 


2.9 


2.1 


2.4 


1.8 


3.7 


Hispanic 


1,229 


8.7 


7.2 


8.1 


11.0 


3.4 


6.3 


2.0 


2.2 


2.1 


3.3 


1.5 


3.4 


Puerto Rican 


753 


5.0 


8.1 


4.8 


10.1 


3.0 


4.8 


1.2 


1.6 


1.1 


2.0 


1.5 


2.4 


English Language 
Dominance 


English 


35,661 


9.9 


9.7 


10.4 


10.0 


5.1 


5.2 


2.5 


2.5 


2.5 


2.5 


2.5 


2.5 


Other 


2,036 


6.7 


12.1 


6.1 


12.3 


4.4 


4.8 


1.6 


3.3 


1.1 


3.3 


2.3 


2.8 


English Fluency 


Fluent 


36,671 


9.8 


9.9 


10.2 


10.1 


5.1 


5.2 


2.5 


2;5 


2.5 


2.6 


2.5 


2.5 


Not Fluent 


348 


3.2 


14.1 


2.6 


12.9 


3.7 


4.0 


1.2 


3.4 


.3 


4.6 


2.0 


2.6 


LSAT Total Score 


120 - 146 


13,813 


4.9 


9.2 


4.3 


12.2 


3.2 


6.1 


.9 


2.3 


.5 


2.9 


1.5 


2.8 


147-154 


13,174 


11.5 


12.6 


11.2 


11.3 


6.5 


5.9 


2.8 


3.3 


2.6 


3.7 


3.2 


3.1 


155 - 180 


12,543 


13.3 


7.5 


16.0 


6.2 


5.8 


3.2 


4.0 


1.8 


4.6 


1.0 


2.9 


1.4 


Number of LSATs 
Taken 


1 


29,035 


10.6 


8.8 


11.1 


9.0 


5.0 


5.0 


2.8 


2.2 


2.8 


2.2 


2.5 


2.4 


2 


8,675 


7.5 


12.4 


8.4 


12.6 


5.6 


5.3 


1.8 


3.2 


1.9 


3.5 


2.7 


2.6 


3-18 


1,820 


7.3 


12.6 


6.4 


13.7 


4.4 


6.0 


15 


3.7 


.9 


3.5 


2.1 


3.2 
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Table 7 (continued) 



Significant Difference Rare Difference 

ARVS.RC® ARvs.LRb RCvs.LR*= ARvs.RC“ ARvs.LR*’ RC vs. LR*= 
Subgroup N (-%) (+%) (-%) (+%) (-%) (+%) (-%) (+%) (-%) (+%) (-%) (+%) 



LSAT Preparation 
Yes 1,648 

No 34,668 

Number of 



LawSchool 
Applications 
1 -2 


9,617 


3-6 


10,063 


7-129 


8,319 


Selectivity of 
Law Schools 
16-36 


7,511 


37-38 


8,044 


39-45 


11,225 



13.4 


7.2 


14.1 


7.3 


9.5 


9.9 


10.1 


10.2 



11.2 


7.1 


12.0 


8.2 


9.9 


10.3 


9.8 


10.3 


7.8 


12.5 


9.1 


11.8 



8.4 


10.8 


;8.9 


11.3 


9.7 


10.9 


10.0 


11.1 


0.8 


8.8 


11.7 


8.4 



3.8 


5.3 


3.9 


1.2 


5.2 


5.1 


2.5 


2.6 



4.7 


5.5 


3.0 


1.5 


5.0 


5.4 


2.6 


2.7 


6.0 


4.5 


1.6 


3.3 



5.2 


5.7 


2.1 


2.7 


5.0 


5.3 


2.4 


2.8 


5.4 


4.7 


2.8 


2.1 



2.8 


1.8 


1.6 


2.7 


2.4 


2.6 


2.6 


2.5 



3.2 


1.8 


2.3 


2.6 


2.4 


2.6 


2.5 


2.6 


1.7 


3.2 


2.8 


2.2 



2.3 


2.9 


2.8 


2.6 


2.2 


2.9 


2.3 


2.7 


2.7 


2.0 


2.5 


2.2 



® Analytical Reasoning vs. Reading Comprehension 
Analytical Reasoning vs. Logical Reasoning 
‘ Reading Comprehension vs. Logical Reasoning 



Significant and rare differences. The frequencies of a significant and rare difference for each pair of subscores 
in the total sample are reported in Table 5; the frequency of positive and negative differences was identical. The 
total number of significant differences was 5.0% for each pair of subscores. The frequencies for the number of sig- 
nificant and rare differences for the three pairs of subscores in the total sample are reported in Table 6; 11.9% of 
the sample had one or more significant and rare differences, and the mei\ was .15 (SD = .44). 

The frequencies of a significant and rare difference for each pair of subscores in the subgroups are reported 
in Table 7, and the analyses are summarized in Table 8. Two differences within the subgroups in these 
variables were significant, statistically and practically. The same variable. Analytical Reasoning vs. Logical 
Reasoning, was significant for age and the LSAT total score. For age, the largest discrepancy involved more 
negative differences (favoring Logical Reasoning) for older examinees. For tire LSAT total score, the largest 
discrepancy involved fewer negative differences for low-scoring examinees. 

The mean number of sigiuficant and rare differences for the three pairs of subscores for the subgroups are re- 
ported in Table 9, and the analyses of variance for the subgroups are summarized in Table 10. None of the 
differences within subgroups was significant, statistically and practically. 

In brief, significant and rare differences in the pairs of subscores were uncommon, but about a tenth of the 
examinees were affected. These differences varied with the LSAT total score and age. 

Intercorrelations 

Observed differences. In the total sample, the observed difference variable for Analytical Reasoning vs. 
Reading Comprehension correlated .74, with the corresponding Analytical Reasoning vs. Logical Reasoning 
variable and -.47 with the Reading Comprehension vs. Logical Reasoning variable, and the Analytical Rea- 
soning vs. Logical Reasoning variable correlated .24 with the Reading Comprehension vs. Logic^ Reasoning 
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Table 8 

Chi Square Analyses of Subgroup Differences in Significant and Rare Differences for Each Pair of LSAT Subscores 



Subscore Pair 


Significant Difference 


Rare Difference 






w 




w 






Sex (df=2) 






Analytical Reasoning vs. 
Reading Comprehension 


47.44** 


.03 


16.12** 


.02 


Analytical Reasoning vs. 
Logical Reasoiung 


266.45** 


.08 


117.16** 


.05 


Reading Comprehension vs. 
Logical Reasorung 


106.80** 


.05 


51.42** 


.04 






Age (df=4) 






Analytical Reasoning vs. 
Reading Comprehension 


781.25** 


.14 


308.36** 


.09 


Analytical Reasoning vs. 
Logical Reasoning 


699.30** 


.13 


366.59** 


.10 


Reading Comprehension vs. 
Logical Reasoning 


64.14** 


.04 


54.35** 


.04 






Ethiucity (df=14) 






Airalytical Reasoning vs. 
Reading Comprehension 


204.58** 


.07 


79.34** 


.04 


Airalytical Reasoning vs. 
Logical Reasoning 


322.57** 


.09 


165.18** 


.06 


Reading Comprehension vs. 
Logical Reasoning 


98.60** 


.05 


69,02** 


.04 




English Language Dominance (df=2) 






Analytical Reasoning vs. 
Reading Comprehension 


31.12** 


.03 


11.50** 


.02 


Analytical Reasoning vs. 
Logical Reasoning 


46.34** 


.04 


21.63** 


.02 


Reading Comprehension vs. 
Logical Reasoning 


2.70 


.01 


1.39 


.01 




English Fluency (df=2) 






Analytical Reasoning vs. 
Reading Comprehension 


21.84** 


.02 


3.69** 


.01 


Analytical Reasoning vs. 
Logical Reasoning 


23.50** 


.03 


12.19** 


.02 


Reading Comprehension vs. 
Logic^ Reasoning 


2.44 


.01 


.38 


.00 
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Table 8 (continued) 



Subscore Pair 


Significant Difference 


Rare Difference 






w 




w 




LSAT Total Score (df=4) 






Anal 3 ^cal Reasoning vs. 
Reading Comprehension 


795.83** 


.14 


321.09** 


.09 


Anal 3 dical Reasoning vs. 
Logical Reasoning 


1,188.77** 


.17 


650.81** 


.13 


Reading Comprehension vs. 
Logical Reasoning 


304.93** 


.09 


173.45** 


.07 




Nvunber of LSATs Taken (df=4) 






Anal 3 dical Reasoning vs. 
Reading Comprehension 


177.50** 


.07 


72.34** 


.04 


Anal)rtiCcd Reasoning vs. 
Logical Reasoning 


191.46** 


.07 


96.26** 


.05 


Reading Comprehension vs. 
Logicd Reasoning 


11.05 


.02 


8.24 


.01 




LSAT Preparation (df=2) 






Anal)rtical Reasoning vs. 
Reading Comprehension 


35.52** 


.03 


24.93** 


.03 


Anal)rtical Reasoning vs. 
Logical Reasoning 


37.62** 


.03 


5.58 


.01 


Reading Comprehension vs. 
Logical Reasoning 


6.38 


.01 


6.45 


.01 




Number of Law School Applications fdf=4) 






Analytical Reasoning vs. 
Reading Comprehension 


192.24** 


.08 


99.37** 


.06 


Analytical Reasoning vs. 
Logical Reasoning 


104.89** 


.06 


73.33** 


.05 


Reading Comprehension vs. 
Logical Reasoning 


26.72** 


.03 


9.27 


.02 




Selectivity of Law Schools (df=4) 






Anal)rtiCcd Reasoning vs. 
Reading Comprehension 


56.00** 


.05 


22.06** 


.03 


Anal)rtical Reasoning vs. 
Logical Reasoning 


88.61** 


.06 


27.60** 


.03 


Reading Comprehension vs. 
Logical Reasoning 


10.12 


.02 


8.30 


.02 


Note. Aw of .10 of a "small" 


effect (Cohen, 1988). 













1 



Tables 

Means and Standard Deviations of Significant and Rare Differences for all Pairs of LSAT Subscores for Subgroups 



Subgroup 


N 


Significant Difference 
Mean SD 


Rare Difference 
Mean SD 


Male 


21,762 


Sex 

51 .77 


.16 


.44 


Female 


17,622 


.48 .75 


.14 


.43 


17-22 


14,202 


Age 

50 .76 


.16 


.44 


23-26 


11,910 


50 .76 


.15 


.43 


27-72 


13,397 


50 .76 


.15 


.43 


American Indian 


276 


Ethnicity 

55 .78 


.14 


.42 


Asian 


2,377 


55 .78 


.17 


.47 


Black 


4,454 


58 .68 


.09 


.34 


Canadian Aboriginal 
and Other 


1,139 


.48 .75 


.14 


.43 


Caucasian 


28,503 


52 .77 


.16 


.45 


Chicano 


626 


.49 .76 


.15 


.44 


Hispanic 


1,229 


.45 .73 


.15 


.43 


Puerto Rican 


753 


56 .68 


.10 


55 


English 


35,661 


English Language Dominance 
50 .76 


.15 


.44 


Other 


2,036 


.46 .74 


.14 


.43 


Fluent 


36,671 


English Fluency 
50 .76 


.15 


.44 


Not Fluent 


348 


.41 .75 


.14 


.43 


120-146 


13,813 


LSAT Total Score 
.40 .69 


.11 


57 


147-154 


13,174 


59 .81 


.19 


.48 


155 - 180 


12,543 


52 .78 


.16 


.45 


1 


29,035 


Number of LSATs Taken 
50 .76 


.15 


.43 


2 


8,675 


52 .77 


.16 


.44 


3-18 


1,820 


50 .76 


.15 


.44 
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Table 9 (continued) 



Subgroup 


N 


Significant Difference 
Mean SD 


Rare Difference 
Mean SD 






LSAT Preparation 






Preparation 


34,668 


.50 


.76 


.15 


.44 


No Preparation 


1,648 


51 


.75 


.14 


.41 




Number of Law School Applications 






1-2 


9,617 


.49 


.75 


.14 


.43 


3-6 


10,063 


51 


.77 


.15 


.44 


7-129 


8,319 


.52 


.77 


.15 


.43 






Selectivity of Law Schools 






16-36 


7,511 


50 


.76 


.15 


.44 


37-38 


8,044 


52 


.77 


.15 


.44 


39-45 


11,225 


50 


.76 


.14 


.43 



variable. In short, the variables for pairs of subscores that had Analytical Reasoning in common were appre- 
ciably related, but the other combinations of variables were minimally associated. 

Significant differences. In the total sample, the significant difference variable for Analytical Reasoning vs. 
Reading Comprehension correlated 51 with the corresponding Analytical Reasoning vs. Logical Reasoning 
variable and -.27 with the Reading Comprehension vs. Logical Reasoning variable, and the Analytical Rea- 
soning vs. Logical Reasoning variable correlated .13 with the Reading Comprehension vs. Logicd 
Reasoning. Thus all the variables were minimally related, apart from an appreciable association for Analyti- 
cal Reasoning vs. Reading Comprehension with Anal)riical Reasoning vs. Logical Reasoning. 

Significant and rare differences. In the total sample, the significant and rare difference variable for Analyti- 
cal Reasoning vs. Reading Comprehension correlated .38 with the corresponding Analytical Reasoning vs. 
Logical Reasoning variable and .17 with the Reading Comprehension vs. Logical Reasoning variable, and 
the Analytical Reasoning vs. Logical Reasoning variable correlated .06 with the Reading Comprehension vs. 
Logical Reasoning variable. Hence, all the variables were miiunrally related, except for an appreciable associ- 
ation for Analytical Reasoning vs. Reading Comprehension with Analytical Reasoning vs. Logical Reasoning. 

Reliability 

Observed differences. The correlations between the observed difference variables for the corresponding 
pairs of half subscores in the total sample were .45 for Analytical Reasoning vs. Reading Comprehension, .46 
for Analytical Reasoning vs. Logical Reasoning, and .23 for Reading Comprehension vs. Logical Reasoning. 
Corrected for double length, the correlations were .62, .63, and .37, respectively. Hence, the reliability was ap- 
preciable for two of the three variables. 

Significant differences. The correlations between the significant difference variables for the corresponding 
pairs of half subscores in the total sample were .26 for Analytical Reasoning vs. Reading Comprehension, .25 
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Table 10 

Analyses of Variance of Subgroup Differences in Significant and Rare Differences for All Pairs of LSAT Subscores 







Significant Difference 


Significant Difference 


Source 


df 


Mean 

Square F 


r\ 


Mean 

Square 


F 


r\ 






Sex 










Between 


1 


8.45 14.54** 


.02 


1.99 


10.46** 


.02 


Within 


39,382 


.58 




.19 










Age 










Between 


2 


.07 .12 


.00 


.36 


1.89 


.01 


Within 


39,506 


.58 




.19 










Ethnicity 










Between 


7 


14.17 24.47** 


.07 


2.62 


13.79** 


.05 


Within 


39349 


.58 




.19 










English Language Dominance 










Between 


1 


2.95 ' 5.08 


.01 


.08 


.41 


.00 


Within 


37,695 


.58 




.19 










English Fluency 










Between 


1 


3.29 5.65 


.01 


.03 


.17 


.00 


Within 


37,017 


.58 




.19 










LSAT Total Score 










Between 


2 


126.41 219.75** 


.10 


,20.88 


108.96** 


.07 


Within 


39327 


.58 




.19 










Number of LSATs Taken 










Between 


2 


1.72 2.96 


.01 


33 


1.74 


.01 


Within 


39327 


.58 




.19 










LSAT Preparation 










Between 


1 


.18 .31 


.00 


32 


1.16 


.01 


Within 


36314 


.58 




.19 










Number of Law School Applications 








Between 


2 


1.95 3.35 


.02 


34 


134 


.01 


Within 


27,996 


.58 




.19 










Selectivity of Schools 










Between 


2 


1.13 1.94 


.01 


30 


138 


.01 


Within 


26,777 


.58 




.19 







Note. An T| of .10 is a "small" effect (Cohen, 1988). 
**^<.01. 
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for Analytical Reasoning vs. Logical Reasoning, and .09 for Reading Comprehension vs. Logical Reasoning. 
The correlation between the number of significant differences for the corresponding pairs of half subscores 
was .17 for the total sample. Thus, the reliability was minimal for all the variables. 



Significant and rare differences. The correlations between the significant and rare difference variables for the 
corresponding pairs of half subscores in the total sample were .16 for Analytical Reasoning vs. Reading Compre- 
hension, .17 for Analytical Reasoning vs. Logical Reasoning, and .07 for Reading Comprehension vs. Logical 
Reining. The correlation between the number of sigruficant and rare differences for the corresponding pairs of 
half subscores was .13 in the total sample. In short, the reliability was minimal for all the variables. 



Discussion 

A central finding is that subscore differences on the LSAT, often very substantial ones, were frequent. Statisti- 
cally significant subscore differences were also very common, affecting about a third of the examinees. 
Substantively, this finding suggests that real differences exist in these examinees' performance on the LSAT 
subtests, reflecting variation in their development of the abilities tapped by the subtests. A similar phenome- 
non has been repeatedly observed in intelligence tests (Chatman, Reynolds, & Willson, 1984; Kaufman, 

1976a, 1976b; Matarazzo, Daniel, Prifitera, & Herman, 1988; Matarazzo & Herman, 1985; McLean, Kaufman, 
& Reynolds, 1989; Rosenthal & Kemphaus, 1988). For example, 37.8% of the standardization sample for the 
WAIS-R (1,880 late adolescents and adults) had a significant difference (.05 level, two-tail) of 10 IQ points be- 
tween their Verbal and Performance IQs (Matarazzo & Herman, 1985). The ubiquity of such discrepancies 
has led to the suggestion that differences in how people manifest intelligence are the norm rather than the 
exception (Kaufman, 1990). 

Statistically significant but rare differences were necessarily less conunon, but affected over a tenth of the ex- 
aminees. These differences have the greatest potential importance because of their infrequency, for they 
represent real and highly imusual differences in examinees' patterns of abilities. 

A related finding of some interest is that these trends in prevalence generally did not vary with the 
examinees' sex, ethnicity, familiarity with the LSAT, or the number or selectivity of the law schools to which 
the examinees were applying. This absence of associations between subscore discrepancies and sex and 
ethnicity is paralleled by generally similar findings for intelligence tes^ts (Chatman et al., 1984; Kaufman, 
1976a, 1976b; Matarazzo, Bomstein, McDermott, & Noonan, 1986; Matarazzo et al., 1988; McLean et al., 

1989). 

The connections that emerged between subscore discrepancies and the LSAT total score and age primarily re- 
flected the deviantly poor performance on the Analytical Reasoning subtest by examinees who had higher 
total scores on the test or were older. 

The greater discrepancies for high scorers accords with the intelligence test research, which consistently 
finds the highest incidence of subscore discrepancies for such examinees (Kaufman, 1976a, 1976b; Matarazzo 
& Herman, 1985; Matarazzo et al., 1988; Matarazzo & Prifitera, 1989; McLean et al., 1989). For instance, in the 
WAIS-R standardization sample, 47.5% of examinees with a full-scale IQ of 120 and above had a significant 
difference of 10 points in their Verbal and Performance IQs, in contrast to 37.8% of the total sample 
(Matarazzo & Herman, 1985). This association has been interpreted as a manifestation of greater differenta- 
tion of abilities for the most intelligent individuals (Matarazzo et al., 1988). 

The greater discrepancies for older examinees disagrees with the research on intelligence tests, which gener- 
ally finds no association between age and subscore differences (Qiatman et al., 1984; Kaufman, 1976b; 
Matarazzo & Herman, 1985; Matarazzo et al., 1988; McLean et al., 1989). This inconsistency may be attribut- 
able to the self-selected character of the LSAT examinee population that produces systematic differences in 
the backgrotmds and experiences of older and younger test takers not fotmd in the representative cross-sec- 
tions of the general population used in the intelligence test work. Follow-up research into the liiJc between 



O 

ERIC 



22 



age and performance on the Analytical Reasoiring subtest might be informative, along the lines of previous 
studies of gender and ethnic group differences in the validity of the test's items and scores (Wightman & 
Muller, 1990a, 1990b). 

Subscore differences are an inevitable consequence when the subtest scores are not perfectly correlated 
(Cahan & Cohen, 1988; Matarazzo, 1972; Matarazzo & Herman, 1985). In fact, significant (and significant 
and rare) differences were most prevalent in this study for pairs of subscores that included Analytical Rea- 
soning, and this subtest had substantially lower correlations with the other subtests (r = J1 with Reading 
Comprehension and r = .59 with Logical Reasoning vs. r = .76 between Reading Comprehension and Logical 
Reasoning in the total sample). This same differentiation between the Analytical Reasoning subtest and the 
other subtests was also observed in a factor analytic study of the previous version of the LSAT (Camilli, 
Wang, & Fesq, 1992). Two factors were identified, one defined by Analytical Reasoning items, and the other 
by Reading Comprehension and Logical Reasoning items. 

Any operational use of these discrepancy variables hinges on their reliability. Two of the three observed dif- 
ference measures had appreciable reliability (Reading Comprehension vs. Logical Reasoning was an 
exception), about .6, similar to the reliability of observed difference measures for intelligence tests (Feingold, 
1984; McNemar, 1957). For example, on the WAIS-R, the average reliabilities were .81 for the difference be- 
tween Verbal and Performance IQs and .65 for the difference between subtest scores (Feingold, 1984). 

But the reliability was minimal for all of the more important discrepancy measures for significant or signifi- 
cant and rare differences, even taking into account the lower-bound nature of these reliability estimates. The 
unreliability of these measures clearly precludes their use in interpreting the test performance of individual 
examinees. Parallel data are unavailable on the reliability of these measures for intelligence tests, but it 
seems highly likely that their reliability is equally low. Indeed, a study of a somewhat similar measure for 
the WAIS-R (number of subtest scores significantly different from the examinee's mean subtest score) found 
its reliability was only .28 (Matarazzo et aL, 1988). ' 

The minimal reliability of these two kinds of discrepancy measures may, of course, contribute to their sparse 
associations with the subgroup variables. However, the much more reliable observed difference measures 
were also generally uirrelated to the subgroup variables, suggesting that this absence of differences in preva- 
lence vydthin subgroups is an inherent feature of all these measures. 

Although subscore discrepancies appear to have no viable role to play in interpreting examinees' LSAT per- 
formance, the distinct possibility remains that other approaches which capitalize on the test's 
multidimensionality, such as using the subscores in prediction systems (Camilli et al., 1992), may yet prove 
to be useful. The potential value of the LSATs multidimensionality merits serious attention. 
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background data for examinees, Ludent C. Chan for computer programming, and Isaac Bejar and Philip K. 
Oltman for reviewing a draft of this rep>ort. 



Endnotes 

1 "Any preparation" consisted of any of the following responses: Studying the same questions in the Law 
Services Information Book: taking the sample test in the Law Services Information Book: working 
through The Official LSAT PrepTest(s). The Offidal LSAT PrepBook. or The Offidal LSAT PrepKit: using a 
book on how to prepare for the LSAT not published by Law ^rvices; attending a commercial test prepara- 
tion or coaching course; attending a test preparation or coaching comse offered through an 
undergraduate institution; self-study; and other preparation. 

2 Data for this variable were only available for examinees applying to 20 or fewer law schools. 
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